PI13 planning.
DAC week. Rucio development. I&P.
I&P stuff, e.g. Dask.
Cloud down. Watching some videos about OAuth2.
Rucio meeting.
Watching deletions..
[user@client-ska-deployment-67bd568548-78wpf ~]$ rucio list-rse-usage IDIA
USAGE:
------
rse_id: 870cfbf4e1444750a7e949cd7e079258
rse: IDIA
source: rucio
used: 4.499 TB
files: 32638
updated_at: 2021-11-17 07:33:07
------
[user@client-ska-deployment-67bd568548-78wpf ~]$ rucio list-rse-usage AARNET_PER
USAGE:
------
rse_id: 2ca1e7eaa08c4f5c9fd26578e4714bbe
rse: AARNET_PER
source: rucio
used: 798.422 GB
files: 7198
updated_at: 2021-11-17 07:32:06
------[user@client-ska-deployment-67bd568548-78wpf ~]$ rucio list-rse-usage AARNET_PER
USAGE:
------
rse_id: 2ca1e7eaa08c4f5c9fd26578e4714bbe
rse: AARNET_PER
source: rucio
used: 575.773 GB
files: 5415
updated_at: 2021-11-17 08:49:14
------
[user@client-ska-deployment-67bd568548-78wpf ~]$ rucio list-rse-usage IDIA
USAGE:
------
rse_id: 870cfbf4e1444750a7e949cd7e079258
rse: IDIA
source: rucio
used: 4.330 TB
files: 31159
updated_at: 2021-11-17 08:53:39
------[user@client-ska-deployment-67bd568548-78wpf ~]$ rucio list-rse-usage IDIA
USAGE:
------
rse_id: 870cfbf4e1444750a7e949cd7e079258
rse: IDIA
source: rucio
used: 3.114 TB
files: 23009
updated_at: 2021-11-17 11:35:41
------
[user@client-ska-deployment-67bd568548-78wpf ~]$ rucio list-rse-usage AARNET_PER
USAGE:
------
rse_id: 2ca1e7eaa08c4f5c9fd26578e4714bbe
rse: AARNET_PER
source: rucio
used: 95.102 GB
files: 666
updated_at: 2021-11-17 11:34:29
------[user@client-ska-deployment-67bd568548-78wpf ~]$ rucio list-rse-usage AARNET_PER
USAGE:
------
rse_id: 2ca1e7eaa08c4f5c9fd26578e4714bbe
rse: AARNET_PER
source: rucio
used: 2.100 MB
files: 24
updated_at: 2021-11-17 19:33:45
------
[user@client-ska-deployment-67bd568548-78wpf ~]$ rucio list-rse-usage IDIA
USAGE:
------
rse_id: 870cfbf4e1444750a7e949cd7e079258
rse: IDIA
source: rucio
used: 2.100 MB
files: 21
updated_at: 2021-11-17 19:32:44
------Used amended script to add rules to IDIA. Removed about 4Tb of files after 3 hours.
Latest:
[user@client-ska-deployment-67bd568548-78wpf ~]$ rucio list-rse-usage IDIA
USAGE:
------
rse_id: 870cfbf4e1444750a7e949cd7e079258
rse: IDIA
source: rucio
used: 7.366 TB
files: 54353
updated_at: 2021-11-16 17:35:50
------
[user@client-ska-deployment-67bd568548-78wpf ~]$ rucio list-rse-usage AARNET_PER
USAGE:
------
rse_id: 2ca1e7eaa08c4f5c9fd26578e4714bbe
rse: AARNET_PER
source: rucio
used: 3.398 TB
files: 28739
updated_at: 2021-11-16 17:32:07
------Submitted PR for helm chart to be changed: https://github.com/rucio/helm-charts/pull/72
Watching https://github.com/rucio/rucio/issues/4949
Created script to find replicas without rules (consequence of rucio uploading a data directory is that it doesn't create rules):
from rucio.client import Client
class bcolors:
HEADER = '\033[95m'
OKBLUE = '\033[94m'
FAIL = '\033[91m'
ENDC = '\033[0m'
BOLD = '\033[1m'
client = Client()
scope = 'testing_long_haul'
rse = 'AARNET_PER'
wipe = True
limit = 9999999
n_dids = sum(1 for _ in client.list_dids(scope=scope, filters={'type': 'DATASET'}, recursive=False))
total_bytes_unaccounted = 0
for idx, name in enumerate(client.list_dids(scope=scope, filters={'type': 'DATASET'}, recursive=False)):
if idx == limit:
break
did = '{}:{}'.format(scope, name)
found_replica = None
for replica in client.list_dataset_replicas(scope=scope, name=name, deep=True):
if replica['rse'] == rse:
found_replica = replica
found_rule = None
if found_replica:
for rule in client.list_did_rules(scope=scope, name=name):
if rule['rse_expression'] == rse:
found_rule = rule
if found_replica and found_rule:
print(bcolors.BOLD, '{}/{}'.format(idx+1, n_dids), bcolors.ENDC, did, bcolors.HEADER, '{}G'.format(total_bytes_unaccounted/10**9), bcolors.OKBLUE, "OK", bcolors.ENDC, found_replica, found_rule, bcolors.ENDC)
if wipe:
try:
client.update_replication_rule(found_rule['id'], options={'lifetime': 60})
except:
continue
elif found_replica and not found_rule:
total_bytes_unaccounted += found_replica['bytes']
print(bcolors.BOLD, '{}/{}'.format(idx+1, n_dids), bcolors.ENDC, did, bcolors.HEADER, '{}G'.format(total_bytes_unaccounted/10**9), bcolors.FAIL, "NOK", bcolors.ENDC, found_replica)
if wipe:
try:
client.add_replication_rule([{'scope': scope, 'name': name}], 1, rse, lifetime=60)
except:
continue
else:
print(bcolors.BOLD, '{}/{}'.format(idx+1, n_dids), bcolors.ENDC, did, bcolors.HEADER, '{}G'.format(total_bytes_unaccounted/10**9), bcolors.OKBLUE, "PASS", bcolors.ENDC, found_replica)and to plot:
from collections import OrderedDict
import datetime
import matplotlib
import numpy as np
import pylab as plt
files = ['IDIA', 'AARNET_PER']
fig = plt.figure(figsize=(10, 8))
for fi in files:
dates = []
sizes = []
with open(fi) as f:
for line in f:
status = line.split()[7]
if status == 'NOK':
replica = eval('{' + line.split('{')[1].split('}')[0] + '}')
dates.append(replica['created_at'])
sizes.append(replica['bytes']/1E12)
dates = np.array(dates)
sizes = np.array(sizes)
dates_ordered = np.sort(dates)
sizes_ordered = sizes[np.argsort(dates)]
sizes_ordered_cumsum = np.cumsum(sizes_ordered)
plt.plot(dates_ordered, sizes_ordered_cumsum, label=fi)
plt.legend()
plt.xticks(rotation=50)
plt.ylabel("usage (Tb)")
plt.savefig("tst.png", dpi=100)Will need to add rules for these replicas.
Looked at Rook operator for managing Ceph.
CephCluster stuck in Connecting state on rucio-dev.
Rucio PR merged for 1.27 in November. Had a little look at how to extend this for custom metadata.
Fixed prod (ran out of space provisioned by NFS due to database backups). Added following to rucio-worker nodes root crontab on both dev and prod (where the NFS volume is attached and served from):
/usr/bin/find /mnt/volume1/postgres-db-backups/ -name "*.sql" -type f -mtime +7 -exec rm -f {} \;Rucio meeting.
Reading around Ceph, specifically using Rook as an operator (creating pools).
Watching some tutorials on flux and OAuth2/OIDC.
Added src-dmz as nameserver.
To use, install resolvconf and add nameserver to tail:
sudo apt-get install resolvconf
sudo vi /etc/resolvconf/resolv.conf.d/tail then restart:
sudo resolvconf -uAdded ceph yamls to ska-rucio-prototype/dev
Had a look (made deletion plots) at reaper for IDIA - looks OK?
Now Rucio metadata component lead.
Tidying up bits.
More Ceph work, added replication throughputs to dashboards from custom event database.
Fixed Rucio PR.
Deployed Ceph cluster for shared PV provisioning.